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I .  INTRODUCTION 


The  Terminal  Ballistics  Division  of  the  Ballistic  Research  Laboratory 
encountered  the  problem  of  determining  how  thick  the  shielding  should  be 
between  rounds  of  ammunition  stored  in  a  storage  area  to  prevent  round- 
to-round  propagation  from  an  initial  explosion.  Vulnerability  analysis 
indicated  that  the  probability  of  survival  of  the  storage  area  would 
drastically  decrease  with  an  increase  in  the  number  of  rounds  exploding. 

Prior  testing  has  shown  that  shielding  material  placed  between  rounds 
could  prevent  neighboring  rounds  from  exploding.  Due  to  space  limitations 
in  the  storage  area,  it  was  desired  to  keep  the  shielding  thickness  to  a 
minimum  and  simultaneously  minimize  the  probability  of  round-to-round 
propagation  from  the  initial  explosion. 

It  was  decided  that  the  specific  objective  of  the  test  would  be  to  find 
the  shielding  thickness  needed  to  be  90%  confident  that  the  probability  of  a 
neighboring  round  exploding  is  less  than  0.1. 

The  problem  appeared  to  fit  into  the  category  of  extreme  value  quantal 
response  problems.  The  stimulus  is  characterized  by  a  variable  X,  in  this 
case  the  thickness  of  the  shielding  which  affects  the  stimulus,  and  the 
probability  of  a  response  associated  with  a  given  X  is  described  by  a  nonresponse 
function  M(X) .  (Usual  notation  has  M(X)  as  the  probability  of  response.  How¬ 
ever,  defining  M(X)  as  a  nonresponse  is  more  natural  for  this  problem.)  This 
function  is  assumed  to  be  nondecreasing  with  increasing  stimulus  levels. 

A  discussion  of  available  designs  and  the  modified  design  chosen 
for  the  experiment  is  contained  in  the  following  chapters. 

II.  AVAILABLE  DESIGNS 

A  nonparametric  approach  was  taken  because  of  the  lack  of  information 
about  the  response  function.  For  a  given  probability  of  an  explosion,  a,  where 
0  <  a  <  1,  we  wish  to  determine  a  value,  X  ,  such  that  M(X  )  =  1  -  a.  As  stated, 

Ut  \X 

the  probability  in  which  we  are  interested  is  a  =  .10,  and  therefore  is  in  the 
tail  of  the  response  distribution.  From  a  review  of  the  available  designs  in 
the  literature  the  only  nonparametric  test  designs  available  for  testing  in  the 
tail  regions  are  the  Alexander  Extreme  Value  Design  and  the  Rothman  Design. 

Of  these,  the  Alexander  Extreme  Value  Design  is  preferred  since  it: 

1)  is  "generally  more  efficient  than  other  available  nonparametric 
designs,  and  is  asymptotically  as  efficient  as  the  best  parametric  stochastic 
approximation  when  distributional  assumptions  are  valid," 

2)  has  significantly  simpler  design  rules  and  analysis  procedures 
than  the  Rothman  Design,  and 

3)  does  not  differ  from  the  Rothman  Design  in  median  required  sample  size. 


*D.  Rothman,  M.  J.  Alexander  and  J.  M.  Zimmerman,  The  Design  and  Analysis  of 
Sensitivity  Experiments,  NASA  CR-62026,  Vol.  I,  p.  74. 
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III.  ALEXANDER  EXTREME  VALUE  DESIGN 


The  Alexander  Extreme  Value  Design  assumes  only  a  nondecreasing  response 
function  as  the  stimulus  increases.* 

A.  Design  Rules 

1)  The  first  test  is  at  shielding  thickness  level  X^  the  a  priori 

best  guess  of  X  . 

ot 

2)  Testing  is  performed  by  alternately  increasing  and  decreasing 
sequences  of  test  levels.  The  test  levels  are  increased  or  decreased  by  a 
step  size  6,  where  6  is  an  estimate  of  M' (X^) .  Terms  such  as  "higher"  and 

"level  above"  refer  to  thicker  shielding  levels,  and  "below"  and  "lowest" 
refer  respectively  to  thinner  and  thinnest  shielding  thickness  levels. 

3)  The  first  sequence  decreases  the  levels  until  a  response 
(explosion)  is  observed. 

4)  The  first  test  of  an  increasing  sequence  is  at  the  level  above 
the  highest  level  at  which  a  response  has  been  observed.  The  increasing 
sequence  ends  at  level  X.  such  that  in  the  corresponding  zero  region*  less 
than  or  equal  to  X.  at  least  N  nonresponses  have  been  observed.  Values 
for  N  can  be  found  from 


(1  _  a)n  =  1  -  P  (1) 

where  N  =  [n]  +  1  and  P  is  some  specified  probability. 

5)  The  first  test  of  a  decreasing  sequence  is  at  the  level  above 
the  highest  level  at  which  a  response  has  been  observed.  If  the  result 
is  a  response,  the  sequence  ends;  otherwise,  one  more  test  at  the  next 
lower  level  is  performed. 

6)  Testing  terminates  when  there  are  three  adjacent  levels,  Xr, 

Xr  +  6  and  Xr  +  26  such  that  at  least  one  response  has  been  observed  at  Xr 
and  none  at  a  higher  level,  and  a  total  of  N  nonresponses  have  been 
observed  at  Xr  +  6  and  X^.  +  26.  (6  is  the  step  size  between  levels.) 

A 

7)  The  maximum  likelihood  estimate  of  X  ,  X  ,  is  found  by  the  method 

a  a. 

of  reversals  and  linear  interpolation  (see  Appendix) . 


Zero  region  -  stimulus  region  above  the  highest  level  at  which  a  response 
has  been  observed. 
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B.  Analysis 


We  are  interested  in  the  a  =  .1  quantile  of  the  response  distribution, 
that  is,  the  value,  X  . ,  at  which  the  probability  of  a  response  is  .1. 

•  -L 

Therefore,  the  probability  of  a  nonresponse  at  the  X  .  quantile  is  (1  -  .1). 

The  probability  of  n  nonresponses,  assuming  the  n  tests  are  independent, 

is  (1  -  .l)n.  The  probability  of  at  least  one  response  out  of  n  tests 

is  1  -  (1  -  .  l)n.  Specifying  the  probability  of  at  least  one  response 
out  of  n  tests  at  the  X  .  quantile  to  be  P  =  .9,  we  have 

.9  =  1  -  (1  -  .l)n. 

This,  with  a  slight  algebraic  manipulation,  is  Equation  (1)  with  a  =  .1 
and  P  =  .9.  Solving,  N  =  [n]  +  1  =  22.  Hence,  we  would  expect  with 
probability  .9  at  least  one  response  out  of  22  tests  at  the  X  ^  quantile. 

If  we  observe  22  nonresponses  at  some  level  X*,  we  can  assume  we  are  not 

at  the  X  .  quantile  and,  in  fact,  the 
•  1 

Prob  {X  1  <  X*}  >  .9  . 

Using  the  above  argument,  we  can  conclude  from  the  Alexander  Extreme 
Value  Design  that  the  level  at  which  the  true  probability  of  response 
is  .1  is  less  than  X^  +  25  with  ninety  percent  confidence.  The  point  estimate 

of  the  X  ..  quantile  can  be  found  using  the  method  of  reversals  outlined  in  the 

Appendix. 

C.  Simulation 


Based  on  engineering  estimates  (judgement)  for  X  ^  and  X  y,.,  a  response 

distribution  was  hypothesized  with  which  to  Monte  Carlo  the  Alexander  Extreme 
Value  Design  for  a  =  .1  and  P  =  .9.  The  response  distribution  assumed  was  the 
cumulative  normal  distribution  with  mean  =  .5  and  variance  =  .14.  (Note, 
however,  that  the  test  design  and  analysis  procedures  are  distribution-free.) 
The  smallest  practical  step  size  of  shielding  thickness  was  1/8  inch. 

Figures  1  and  2  are  examples  of  the  Alexander  Extreme  Value  Design 
Monte  Carloed  to  illustrate  the  design  rules.  Responses  are  denoted  by 
"X"  's  and  nonresponses  by  "0"  '  s.  1^  denotes  the  i-th  increasing  sequence 

and  D.  the  j-th  decreasing  sequence.  The  number  of  rounds  required  (NR), 

J  * 

the  maximum  likelihood  estimate  of  the  .1  quantile  (X  j) ,  and  the 

(Xr  +25)  level  are  given  for  each  simulation. 

Figure  3  shows  the  distribution  of  the  number  of  rounds  required 
for  500  simulations  of  the  above  design.  The  number  of  rounds  required 
is  twice  the  number  of  responses  and  nonresponses  shown  for  each  simulation 
since  a  donor  round  must  be  detonated  for  each  test  round.  The  average 
number  of  rounds  required  to  complete  the  test  was  166,  the  median  was 
164  and  ten  percent  of  the  tests  required  184  rounds  or  more. 
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The  distribution  of  the  maximum  likelihood  estimates  of  X  j  for 
the  500  simulations  is  given  by  the  histogram  in  Figure  4.  The  distribu- 

A 

tion  of  X  ,  is  asymptotically  normal  about  the  true  X  .  quantile  =  7.83. 

The  distribution  generated  by  the  test  data  shown  in  Figure  4  has  a 
mean  of  7.77,  which  is  in  good  agreement  for  500  simulations,  and  is 
approximately  normally  distributed  as  shown  by  the  overlying  normal  curve. 

Figure  5  shows  the  distribution  of  level  Xr  +  26  for  500  simulations. 
This  is  the  level  about  which  we  can  conclude  that  the 

Prob  {X  ,  <  X  +  26}  >  .9. 

.  l  r 


IV.  MODIFICATION  OF  THE  ALEXANDER  EXTREME  VALUE  DESIGN 

The  median  number  of  rounds  required  for  the  Alexander  Extreme 
Value  Design  (EVD) ,  as  described  in  the  previous  section,  was  164  as 
determined  by  the  500  simulations.  Since  the  number  of  rounds  available 
for  testing  was  considerably  smaller,  the  major  objective  in  modifying 
the  Alexander  EVD  was  to  reduce  the  number  of  rounds  required,  while 
maintaining  the  confidence  level  and  the  ability  to  compute  the  point 
estimate  of  the  X  ^  quantile. 

The  Alexander  EVD  requires  that  a  donor  round  be  detonated  for  each 
test.  The  number  of  donors  needed  can  be  reduced  by  using  one  donor  to 
detonate  up  to  four  test  rounds  (acceptors).  Figure  6  shows  the  config¬ 
uration  of  four  acceptors  per  donor.  Steel  shielding  will  be  placed 
between  acceptors,  as  shown  by  the  dotted  lines,  if  interaction  between 
acceptors  is  observed.  Optimizing  the  number  of  acceptors  per  donor  in 
the  Alexander  EVD  reduces  the  number  of  rounds  required  by  approximately 
33  percent. 

It  was  noticed  that  the  rounds  above  level  X  +  26  were  used  neither  to 

r 

establish  the  confidence  statement,  nor  to  terminate  the  test  design,  nor  to 
compute  the  point  estimate  of  the  X  ^  quantile.  By  limiting  each  increasing 

sequence  above  the  highest  stimulus  level  at  which  a  response  has  been 
observed,  the  rounds  "wasted"  above  level  X  +  26  can  be  eliminated.  There 

is  a  trade-off  in  eliminating  these  rounds  since  the  level  Xr  +  26  can 

change  if  a  response  is  observed  at  a  higher  level.  Therefore,  some  testing 
should  be  above  X^  +  26  until  more  than  half  the  number  of  rounds  required 

to  demonstrate  the  chosen  probability  are  at  levels  X^  +  6  and  Xr  +  26. 

Testing  at  X^  and  below  is  used  in  the  determination  of  the  point  estimate 
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The  following  test  design  is  the  result  of  many  Monte-Carlo  simula¬ 
tions  in  which  different  starting  levels,  number  of  acceptors  per  donor 
and  sequences  of  testing  have  been  tried  in  order  to  minimize  the  required 
number  of  rounds,  yet  retain  the  confidence  level  and  point  estimate  of 
the  X  ..  quantile. 

A.  Modified  Design  Rules 

1)  The  first  test  level  is  X^,  the  best  a  priori  guess  of  Xa- 
6  is  the  step  size  between  levels. 

2)  One  acceptor  per  donor  is  used,  in  a  decreasing  sequence,  until  a 
response  is  observed.  Let  X  be  the  highest  level  at  which  a  response 

is  observed. 

3)  After  the  first  response,  the  number  of  acceptors  per  donor 

in  each  test  is  increased  to  alternately  three  and  then  four.  After  the 
first  response,  three  acceptors  per  donor  are  tested  at  the  next  three  levels 
above  X  .  Then  four  acceptors  per  donor  having  shielding  at  levels  Xr 
and  the  next  three  higher  levels  are  tested. 

4)  If  another  response  is  observed  at  a  higher  level,  it  becomes 
X  ,  and  testing  continues  alternating  three  and  then  four  acceptors  per 
donor  until  at  least  12  (more  than  half  the  required  22)  nonresponses 
have  been  observed  at  the  two  levels  immediately  above  X^. 

5)  When  at  least  12  nonresponses  have  occured  at  Xr  +  6  and  X^  +  26, 

the  number  of  acceptors  per  donor  is  reduced  to  alternately  two  above  Xr  and 
then  three,  starting  at  Xr,  for  the  remainder  of  the  test.  ' 

6)  Testing  terminates  when  at  least  N  (22)  nonresponses  have  been 
observed  at  the  two  levels  immediately  above  the  highest  level  at  which 
a  response  has  been  observed. 

B.  Analysis  of  the  Modified  Design 

As  in  the  Alexander  Extreme  Value  Design,  we  have  N  =  22  nonresponses 

at  X  +6  and  X  +  26  and  can  conclude  that  we  are  not  at  level  the  X  , 
r  r  .1 

quantile  and  in  fact, 

Prob  (X  .  <  X  +  26}  >  .9. 

.1  r 

The  point  estimate  can  again  be  found  using  the  method  of  reversals.  There¬ 
fore,  the  changes  in  the  test  design  have  not  affected  the  confidence 
statement  or  the  point  estimate. 

C.  Simulations  Using  Modified  Design 

Using  the  same  response  distribution  that  was  used  when  simulating  the 
Alexander  Extreme  Value  Design,  500  simulations  of  the  modified  design 
were  also  Monte-Carloed. 
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Figures  7  and  8  are  examples  of  the  modified  test  design  illustrating 
the  modified  design  rules.  Again,  responses  are  denoted  by  "X"  ’  s  and 
nonresponses  by  "0”  ’  s.  The  abscissa  represents  individual  tests  rather  than 
sequences  of  tests  as  shown  in  the  Alexander  Extreme  Value  Design. 

Figure  9  shows  the  distribution  of  the  required  number  of  rounds  for 
the  500  simulations.  The  median  number  of  rounds  required  was  67  and  the 
mean  number  of  rounds,  70.  Only  ten  percent  of  the  simulations  required 
93  or  more  rounds. 

The  histogram  in  Figure  10  is  the  distribution  of  the  maximum 
likelihood  estimates  of  X  ^  for  the  500  simulations.  Again,  the  distri¬ 
bution  of  the  maximum  likelihood  estimates  are  asymptotically  normal 
about  the  true  X  ^  quantile  =  7.83.  The  distribution  shown  has  a  mean  of 
8.00,  and  is  approximately  normally  distributed  as  shown  by  the  overlying 
normal  curve.  Figure  12  shows  the  distribution. of  number  of  rounds  required 
for  both  the  Alexander  EVD  and  the  Modified  Alexander  EVD.  The  Modified 
Alexander  EVD  is  on  the  left  and  the  Alexander  EVD  is  on  the  right. 


V.  OTHER  SIMULATION  DISTRIBUTIONS 

All  the  work  thus  far  has  used  a  normal  distribution  as  the  underlying 
distribution  for  generating  the  simulated  data.  In  order  to  investigate  the 
effect  of  different  distributions  on  the  two  designs,  the  X  5  and  X  75 

"guestimates"  were  used  to  approximate  three  more  distributions,  the  exponen¬ 
tial,  the  uniform  and  the  gamma.  The  results  are  summarized  in  Table  1.  It 
should  be  noted  that  the  true  X  ^  quantile  is  different  for  each  distribution. 

However,  both  the  Alexander  and  Modified  Alexander  EVD  give  good  estimates 
of  X  The  Modified  Alexander  EVD  average  estimates  of  X  .  are  slightly 

higher  than  the  Alexander  EVD  in  all  cases  except  for  the  uniform  distribution. 
The  Modified  Alexander  EVD  requires  less  than  half  as  many  samples  as  the 
Alexander  EVD  for  all  the  distributions.  The  average  of  the  90%  upper 
confidence  limit  is  also  given  for  each  distribution.  The  ratio  of  the  mean 
sample  size  required  for  the  modified  Alexander  EVD  to  the  mean  of  the 
Alexander  EVD  is  given  in  the  last  column.  This  ratio  indicates  that  on  the 
average  the  modified  design  requires  50%  or  less  samples  than  the  Alexander 
EVD. 


VI .  SUMMARY 

The  Alexander  EVD  was  modified,  mainly,  by  using  multiple  rounds  per 
test  and  by  limiting  the  number  of  rounds  above  the  highest  response.  These 
changes  resulted  in  a  design  that  required  less  than  half  the  rounds  of  the 
Alexander  EVD  in  the  simulations  performed,  regardless  of  the  simulated  dis¬ 
tribution.  The  sample  size  required  for  Modified  Alexander  EVD  ranged  from 
37%  to  50%  of  that  required  for  the  Alexander  EVD.  The  Modified  Alexander 
EVD  has  simple  design  rules  that  permit  the  estimation  of  an  extreme  value 
of  a  quantile  response  function  and  the, associated  confidence  interval.  It 
should  be  remembered  that  neither  the  experimental  design  nor  the  analysis 
methods  require  the  assumption  of  a  response  distribution.  The  design  is 
distribution-free. 
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APPENDIX 


METHOD  OF  REVERSALS  FOR  SENSITIVITY  DATA 

A.  Method 

The  method  of  reversals  is  a  maximum-likelihood  procedure  for  obtaining 
distribution-free  estimates  of  a  monotone  nondecreasing  response  function. 
The  test  stimulus  levels  are  X. (i  =  l,2,...,k)  and  are  ordered  from 
thickest  to  thinnest  shielding  thickness, 

Xj  >  X2  >  ...  >  Xk  (A.  1) 

A 

If  p^  is  the  estimate  of  the  probability  of  response  at  X^,  and  if  we 

assume  that  the  response  function  is  monotone  nondecreasing,  then  neces¬ 
sarily 

A  A  A 

Pj  ^  P2  ^  ^  Pk*  (A.  2) 

The  algorithm  below  can  be  used  to  find  the  estimates  of  the  response  dis¬ 
tribution  and  their  associated  stimulus  levels. 


1)  Let  X.  (i  =  1,2, ...,k)  be  the  k  stimulus  levels  at  which  data  have 

1  Ak 

been  collected,  where  X1  >  X2  ...  >  X^.  We  wish  to  find  the  estimates,  p^, 
of  the  values  p^^  =  M(X^),  the  response  probabilities  at  the  levels  X^,  which 
satisfy  Equation  A. 2. 

2)  Let  n^  (i  =  l,2,...,k)  be  the  number  of  tests  performed  at  level 

X^  and  f^  (i  =  l,2,...,k)  be  the  number  of  responses  observed  in  the  m  tests. 
Consider  the  sequence 


If  this  sequence  is  nondecreasing,  then  the  estimates  p.  are  simply  given 
by  .  1 


X 1  *  1 4*  ^ 

3)  If  for  some  i,  —  >  -  ,  replace  both  by 

i  ni+l 


*i,i+l  £i  +  f 


i+1 


N 


i,i+l 


n.  +  n.  . 
l  l+l 
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The  new  sequence  is  then 


f.  .  F.  .  .  f.  . 
l-l  1,1+1  1+2 

n.  »  N.  .  .*  n.  * 
l-l  1,1+1  1+2 


If  this  sequence  still  contains  a  reversal,  a  pair  of  consecutive  fractions, 
for  which  the  first  is  greater  than  the  second,  replace  the  pair  with  a 
single  term  as  above.  This  process  is  continued  until  one  obtains  a  non¬ 
decreasing  sequence: 


*1  *2  *3 

V  V  V 


•  •  •  9 


<j> .  f  ^  +  . . .  +  f^+s 

where  —  =  -  for  appropriate  i  and  s. 

n.  n.  +■...+  n. 

J  i  l+S 


4)  The  final  estimates  are  given  by 


Pi  -  , 


)  .  =  — *— 
i+s  n . 
3 


5)  Linear  interpolation  is  used  to  compute  the  values  of  the  response 
function  between  stimulus  levels  tested. 
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B.  Example 


If  the  results  of  the  experiment  were  as  shown  in  Figure  A1  the 
maximum  likelihood  estimate  found  by  the  method  of  reversals  is  as 
follows: 


Shielding  Thickness 


f./n. 
1  1 


i,i+l 

N.  .  . 
1,1+1 


4/8 

1/1 

1.0 

S/8 

0/2  “ 

1/4 

.3 

6/8 

1/2  _ 

3/10 

.3 

7/8 

2/6 

— 

.3 

8/8 

1/9 

.11 

9/8 

0/12 

0 

10/8 

0/10 

0 

The  shielding  thickness  corresponding  to  the  .1  quantile  is  found  by 
linear  interpolation. 


.0l[ 


8/8  =  1 

X.l 

9/8  =1.13 


.11“ 

.( 

.1 


The  shielding  thickness  associated  with  the  .1  quantile  is  1.01  inches. 
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